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We study the exchange interaction and the subsequent collective behavior of magnetic impurities 
embedded in a disordered two-dimensional (2D) helical metal. The exchange coupling follows a 
statistical distribution whose moments are calculated to the lowest order in {pfF) 1 , where pf is 
the Fermi momentum of itinerant electrons and l is the mean free path. We find that i) the first 
moment of the distribution decays exponentially, and ii) the variance of the interaction is long-range, 
however, it becomes independent of the orientation of the localized magnetic moments due to the 
locking between spin and momentum of the electrons that mediate the interaction. As consequence, 
long-range magnetic order tends to be suppressed, and a spin glass phase emerges. The formalism 
is applied to the surface states of a three-dimensional (3D) topological insulator. The lack of a net 
magnetic moment in the glassy phase and the full randomization of spin polarization at distances 
larger than l excludes a spectral gap for surface states. Hence, non-magnetic disorder may explain 
the dispersion in results for photoemission experiments in magnetically-doped topological insulators. 
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A topological insulator is a system that supports 
metallic edge/surface states within the bulk gap, whose 
existence and integrity are protected by time-reversal 
symmetry jf|. In the case of a 3D topological insula¬ 
tor, the surface states disperse as Dirac quasi-particles; 
the minimal Hamiltonian consists in a Bychkov-Rashba 
spin-orbit coupling of the form (h = 1) 

'Hbr =v F (a x p) z , (1) 

where v F is the Fermi velocity of surface electrons and 
er = (cr x , a y ) is a vector of Pauli matrices acting on elec¬ 
tron spin. The inclusion of a mass term in the Hamilto¬ 
nian, Ma z , reflects the breakdown of time-reversal sym¬ 
metry. This may arise as a Zeeman term due to a weak 
magnetic field [2] or the proximity exchange coupling to 
a magnetic thin film [3]. In this scenario, the system be¬ 
comes a 2D quantum Hall liquid, a perfect platform for 
several magneto-electric effects [7]. 

An interesting possibility is the deposition of magnetic 
adatoms, in such a way that, for high enough concentra¬ 
tions, n m , and at low enough temperatures, T, a spon¬ 
taneous ordering opens a gap in the spectrum, being the 
mass M proportional to the net magnetic moment in the 
out-of-plane direction z gHS]. Experimentally, this pos¬ 
sibility does not seem to be completely clear. Angle- 
resolved photoemission (ARPES) and scanning tunnel 
spectroscopy (STS), together with X-ray magnetic cir¬ 
cular dichroism (XMCD) and magneto-transport experi¬ 
ments generate results that are in agreement with a gap 
opening in some cases [5] and gapless spectrum in others 
[5] for similar experimental conditions. In the particular 
case of Fe adatoms deposited directly on the surface of 
Bi 2 Se 3 , for example, a gap of the order of 100 meV at 
T ~ 15 K was reported in an ARPES experiment [ID] , 
whereas later on, another ARPES experiment in similar 



FIG. 1: Sketch of the RKKY interaction mediated by itin¬ 
erant electrons in a disordered helical metal. 


conditions of Fe concentration and even lower temper¬ 
atures reported no gap opening HU. further confirmed 
by STS and XMCD experiments mi. This dispersion in 
experimental results motivates the present study about 
the impact of disorder in the collective behavior of mag¬ 
netic adatoms placed on the surface of a 3D topological 
insulator. 

We consider first the problem of the exchange interac¬ 
tion between localized spins associated to magnetic im¬ 
purities embedded in a weakly disordered helical metal. 
The Hamiltonian reads in general 

H = 'Hu + H-ioc, 

where the first terms refers only to the itinerant electrons 
and the second term describes the coupling with localized 
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spins, 


n loc = J Z J2 ( R ‘) ’ s i + J \\Y S ( R *) ’ Si > 

i£m 26m 

where the sum runs over the magnetic impurities. Here 
S*, Si = (S*,S“) are the spin operators of the impu¬ 
rity and s z (Rj), s(Rj) = (s^ (Ri) ,s y (Ri)) are the spin 
density operators of the metal evaluated at the magnetic 
impurity sites, 

s a (Ri) = ^ d (2) (Ri - r) <r a (r). 

r 

The exchange is taken to be anisotropic due to the 2D 
nature and strong spin-orbit coupling of the system. In¬ 
plane isotropy is assumed, J x = J v = Jy. 

The single-particle Hamiltonian for itinerant electrons 
can be written as / H it = Her + V(r), where the first 
term corresponds to Eq. ([I]), and the second describes 
the effect of disorder, V (r) = nm (r - Ri). Here 
Vimp (r) is the potential created by a non-magnetic impu¬ 
rity or any source of disorder that preserves time-reversal 
symmetry. Since we do not posses a detailed expres¬ 
sion for Vi m p (r) and the distribution of impurities change 
from sample to sample, we employ a statistical descrip¬ 
tion in terms of different disorder realizations forming 
an ensemble of macroscopically identical replicas of the 
system. Assuming that the typical decay length of the 
impurity potential is smaller than the mean separation 
between scattering centers, we consider a Gaussian dis¬ 
tribution of disorder configurations, determined by the 
mean free path £, or equivalently, the scattering time 
r = Z/vf j and characterized by correlators of the form 

(V (rO V (r 2 )) dis = ^ t ^ 2) Oi - r 2 ), 


where 7 is the density of states at the Fermi level, 
7 = pf/(27tuf)- The itinerant electrons mediate 
the Ruderman-Kittel-Kasuya-Yosida (RKKY) interac¬ 
tion m between the localized magnetic moments. For 
a dilute concentration of magnetic impurities this can be 
studied within second order perturbation theory. Then, 
the effective exchange coupling is determined by the 
static spin susceptibility of the helical electron gas. In 
the presence of non-magnetic disorder, this follows a sta¬ 
tistical distribution whose moments can be computed by 
standard diagrammatic techniques. The first moment 
of the distribution is defined as Xa/3 (R = Rj — Ri) = 
(Xa/3 (Ri, Rj)) Ms’ w h ere we emphasize with the notation 
that translation invariance is restored on average. To the 
lowest order in (pf ^) _1 w e can write 


Xa /3 I 


(R) = —— ^^Tr <y a G (iui, R) (jpG (ito, — R) 


where G(iu/,R) = (G (iu, Ri, Rj) ) is the disorder- 
averaged Green operator in the Matsubara frequencies 
domain, and the trace is taken in the spin indices. Here 
we are neglecting vertex corrections, which is strictly 
valid at R = |R| £., whereas the self-energy correc¬ 

tions are introduced by means of r in the self-consistent 
Born approximation. The hierarchy of length scales of 
the problem for which this equation strictly applies is 
Pp 1 C(CH, as illustrated in Fig. [lj This corresponds 
to a classical diffusive regime: helical electrons connect¬ 
ing a pair of localized spins describe a classical diffusion 
path suffering several collisions with non-magnetic scat¬ 
tered. In the T —> 0 limit, the disorder-averaged RKKY 
interaction reads 


(H RKK Y) dis = ~ ^2 V R 2 [ J z sin ( 2 R Pf ) S i s j + J \\ sin ( 2 R Pf ) (R- ■ S;) • (R ■ Sj) - JzJ\\ cos (2 Rp F ) (Si x S j) ±xA 

i,j(z m 

( 2 ) 


J z controls the exchange between the out-of-plane spin 
components, and Jy between the projections along the 
direction linking the two magnetic impurities. The last 
term is a Dzyaloshinskii-Moriya coupling. In the diffusive 
regime, the three components decay exponentially as in 
a conventional 2D electron gas in. This must not be 
interpreted as an exponential suppression of the RKKY 
interaction, but as the result of the randomization of its 
characteristic oscillatory tail m Consequently, higher 
moments of the distribution must be studied. 


The calculation of the second moment of 
the distribution, XapXa'i3' (R = Rj — Ri) = 
(Xa/3 (R*,Rj)Xa'/3'(Rj,Rj)) dis , is analogous to the 
calculation of the universal conductance fluctuations 
in a disordered conductor m- To the lowest order in 
(pf£) i the second moment is given by both diffuson 
and cooperon ladder contributions, see Fig. [2j In the ab¬ 
sence of time-reversal symmetry breaking perturbations 
(excluding the localized spins) both contributions are 
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FIG. 2: Diffuson and cooperon ladder contributions to 

XapXa'p 1 - The latin indices refer to spin components of the 
diffuson/cooperon kernel, and the greek indices label the spin 
operators at the vertex. 


equal. We can write 

(27T^ r ) 2 

XapXa’0' (R) = 22 D » v ~ R ) 

(iwi - *w 2 , -R) TV [oy^avov] Tr [c^oy 073 'oy], 


where the diffuson modes are introduced as 

D^ v (iw,R) = — [viilji Diji'ji (iw,R) . 

Here do corresponds to the identity and the contraction in 
spin indices (latin labels) is assumed. Diagrammatically, 
Diji'j ' corresponds to the boxes in the first diagram of 
Fig- U At this point, it is useful to introduce the matrix 


( Aio 

D()x 

Dq V 

Do z 

\ 

D x o 

Dxx 

D X y 

D xz 


Dy 0 

Dyx 

Dyy 

Dyz 


\ D z0 

D zx 

Dzy 

D zz 

) 


In the diffusive regime, |w| r <C 1, with u> = — w 2 and 

sign(wi) ^ sign(w 2 ) (otherwise, the diffuson ladder is 
zero), the diffuson satisfy the equation 


/ \u)\t — tD\7 2 
0 
0 

V o 


0 

■ |w| r — tZ?V 2 

0 

-\rv F d x 


0 

0 

- |w| t — tDV 2 

-\ TV Fdy 


\ 


0 

brv F d x 
\rv F d y 

|w| r — rUV 2 ) 


D (iuj , r, r') = <5^ (r — r') 


(3) 


r 


where D = 4 VpT is the diffusion constant. The triplet 
modes (p,, v = x,y,z) are coupled by precession terms 
and only the singlet mode D 00 remains gapless . There¬ 
fore, the triplet modes yields to an exponentially van¬ 
ishing contribution to XapXa'p 1 (R), similarly to what 
happens in a 2D electron gas with strong spin-orbit scat¬ 
tering OH- Only the singlet mode contribution survives, 
leading in the T —> 0 limit to 

p 2 

XuPXu'P' (R) = 9^ 7J .4 v2 jj>4 ^ao’ <5/3/3' • (4) 

As it is evident from this result, the RKKY coupling is 
still a long-range interaction, however, the variance of the 
interaction becomes independent of the orientation of the 
localized spins, ((Hrkky) 2 ) ~ SfS 2 . 

\ . / dis J 

Different magnetically ordered phases have been pro¬ 
posed in the literature HH23- In diffusive media, however, 
these phases are expected to be suppressed due to the ex¬ 
ponential decay of the first moment of the distribution for 
the exchange coupling, Eq. since, according to mean 
field arguments, it is this quantity what determines the 
critical temperature below which long-range magnetic or¬ 
der is possible. On the other hand, disorder introduces 
non-negligible fluctuations of the exchange coupling. In 
the particular case of an helical metal, the locking be¬ 
tween momentum and spin of carriers makes the spin 
response independent of the orientation of the localized 


spins, Eq. Q. Due to these fluctuations, we expect that 
localized spins freeze to a non-zero value at low tempera¬ 
tures, but with no spatial correlation between them and 
zero net magnetization. In such a glassy phase, a global 
spectral gap is excluded. 

We focus on the collective behavior of magnetic 
adatoms assuming that their spatial distribution is com¬ 
pletely random. The clean limit with the Fermi level ly¬ 
ing at the Dirac point was analyzed in Ref. [BJ The form 
of the interaction in that case makes the in-plane interac¬ 
tions frustrated. Then, different behaviors are expected 
as a function of S = J\\/J z - For 6 <C 1, a Ising-like fer¬ 
romagnetic phase is expected at low temperatures, with 
magnetization along the out-of-plane axis. Hence, a gap 
in the spectrum of surface states is expected. In the op¬ 
posite limit, i5 > 1 , the frustrated in-plane interactions 
dominate, giving rise to a spin glass separated from the 
Ising ferromagnet by a quantum critical point estimated 
to be located at S c ss 1.3 [ 6 ]. Similarly, fluctuations in 
the exchange interaction due to disorder tend to destroy 
long-range order, and the system freezes to a glass for 
sufficiently low temperatures. 

We analyze in detail the situation when <5 -C 1. The 
spin Hamiltonian reads 

Hrkky = - 22 
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FIG. 3: Phase diagram of magnetic adatoms randomly dis¬ 
tributed over the surface of a 3D topological insulator, show¬ 
ing paramagnetic (PM), ferromagnetic (FM), modified ferro¬ 
magnetic (FM’) and spin glass (SG) behavior. The results 
are based on the SK model for 5 <C 1. 


where the couplings Jy follow a certain statistical distri¬ 
bution. The first and second moments can be written in 
general as 


(^•) dis = ^ 2 E^« R })^( R )- 

{R} 

(( = J zJ2 V (W) XzzXzz (R) • 

{R} 


Here V ({R}) describes the statistical distribution of 
magnetic adatoms over the surface of the topological in¬ 
sulator. We assume for simplicity that they are uniformly 
distributed, so then we can approximate the above equa¬ 
tions by [TSj 


{Jij) 


dis 


A 


d 2 R Xzz (R) 


rim JzPF 0 f 1 

4 v f n \ PF e 


(5) 


and similarly for the second moment, 

( {jij)2 ) dis * / rf2R (r) 

_ nmJjPF f°° J_ = n m J*p 2 F 
127 t 3 v f N J e R 3 247r 3 u| ,£ 2 N 

Here A is the area of the system and N = n m A , the num¬ 
ber of localized spins. Note that the integral in Eq. © 
is infrared divergent, but the result of Eq. Q is strictly 
valid at R £; therefore, there is a natural cut-off for 
this integral determined by the mean free path, which 
allows to estimate Jij ) 2 ^) to the leading order in 
[pF^y 1 . 

We assume that higher moments of the distribution do 
not affect the thermodynamic properties of the system, 
which can be studied within the Sherrington-Kirkpatrick 


(SK) model [H]. Following SK, the saddle-point equa¬ 
tions reads 


to = 

q = 


f°° dz 
— oo V 27T 
00 dz 

— oo v 27T 


1 _2 

2 Z tanh 
z tanh 2 


/ Tfm 

V T 

f Tfm 

V T 


TO + 

TO + 


Tsg 

T 

Tsg 

T 



(7) 

( 8 ) 


where m = is the magnetization 

per spin and q = ( ^ XiGm (8 >Dt / is the Edwards- 
Anderson order parameter ED], and we have introduced 


Tfm 
Tsg = 


iJzPF 


4 vf^b 


and 


JzPF 
27 TVFkB 


'^"m'^'dis 


67T 


_^ j 2 

Here we have taken i ~ n dis , where n^is represents the 
concentration of non-magnetic strong scatterers. 

For low concentration of scatterers the model predicts 
a second order phase transition to a ferromagnetic state 
at T = Tfm cx n m . In the opposite limit, fluctuations in 
the couplings destroy long-range order, and a spin glass 
phase characterized by g / 0 but to = 0 is stabilized 
below Tsg oc y/n m ndi S ■ The phase diagram is shown 
in Fig. [3j Numerical solution of Eqs. 0-© predicts a 
region of the parameter space where the system passes 
from a paramagnetic to a ferromagnetic to a reentrant 
spin glass phase (dashed line in Fig. [ 3 ]) as the temper¬ 
ature is reduced. This is strange since at intermediate 
temperatures, where entropy plays a role, the system is 
ordered, but as T —> 0 one finds that a disordered phase 
is preferred. This is associated to the instability of the 
saddle-point solution of the SK model and the replica 
symmetry breaking. From Parisi’s solution [21| we pre¬ 
dict a transition from the spin glass to a modified fer¬ 
romagnetic phase (sometimes called mixed phase, where 

o 3 

the replica symmetry is broken) at ndis = The 

boundary between this modified ferromagnetic phase and 
a conventional Ising ferromagnet can be estimated from 
the Almeida-Tlrouless line 155]. 

Within this mean-held scenario, the formation of a 
spin glass at temperatures well below Tsg can be seen 
as quenched fluctuations of a mass term in Eq. ([I]). Net 
magnetization is absent beyond the disorder threshold 

Q 3 

ndis = J7r 2 n,n , and the situation resembles the quantum 
percolation picture of the plateau transition of the quan¬ 
tum Hall effect [23], where the mass plays the role of 
the control parameter. These fluctuations are marginally 
irrelevant in the renormalization group sense, whereas 
higher moments are marginal at the tree level, what jus¬ 
tifies our previous assumptions. 

In summary, we have computed the first moments of 
the distribution for the RKKY interaction in a weakly 
disorder helical metal and applied the results to in¬ 
fer some trends in the collective behavior of magnetic 
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adatoms on a 3D topological insulator. For J z ,J\\, 
our analysis reveals that the ensemble of localized spins 
freezes to a spin glass phase above the disorder threshold 

Q 3 

n-dis = J7r 9 nm . In-plane exchange interactions may help 
to stabilize this phase for weaker disorder due to frustra¬ 
tion. The randomization in the orientation of the spins of 
the magnetic adatoms due to the helicity of carriers en¬ 
forces this picture. In the spin glass phase, a spectral gap 
is precluded due to the lack of a net magnetic moment. 
This means that an ARPES experiment would report no 
gap opening in this scenario. Therefore, the presence of 
disorder may explain the dispersion in results for pho¬ 
toemission experiments in magnetically-doped topologi¬ 
cal insulators. Nevertheless, a local probe [53] could test 
the consequences of the breakdown of time reversal sym¬ 
metry due to the freezing of the moments of the magnetic 
adatoms. 

Finally, it is worth to mention that the effect of a time- 
reversal symmetry breaking in the electron gas that me¬ 
diates the interaction is not taken into account in a self- 
consistent way. In a recent calculation, the authors of 
Ref. [53] showed that a gap opening does not change the 
general structure of the RKKY interaction. Similarly, ef¬ 
fects of dissipation can be neglected at first glance based 
on the argument that the RKKY interaction depends on 
electronic states deep inside the Fermi sea, not only at 
the Fermi surface, whereas dissipative phenomena as the 
Kondo effect is purely a Fermi-surface effect [5B| ■ In other 
words, a rearrangement of the low energy part of the 
spectrum does not dramatically affect the RKKY inter¬ 
action. Nevertheless, interesting novel physics beyond 
the scope of this work may appear at low temperatures, 
particularly in the disordered phase. 
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